Papers with task-specific metrics
Machine Translation of Restaurant Reviews: New Corpus for Domain Adaptation and Robustness (D19-56)
Copied to clipboard
Alexandre Berard, Ioan Calapodescu, Marc Dymetman, Claude Roux, Jean-Luc Meunier, Vassilina Nikoulina
| Challenge: | BLEU: MT is a very robust and efficient way to translate user-generated content. |
| Approach: | They propose a task to encourage research on MT robustness and domain adaptation . they ask professionals to translate 11.5k french 4SQ reviews to English . |
| Outcome: | The proposed task improves on the existing MT systems in a real-world scenario . the proposed methods improve translation accuracy and sentiment analysis . |
VizSeq: a visual analysis toolkit for text generation tasks (D19-3)
Copied to clipboard
| Challenge: | Several softwares for text evaluation are available that do not provide detailed examples. |
| Approach: | They propose a visual analysis toolkit for instance-level and corpus-level system evaluation on a wide variety of text generation tasks. |
| Outcome: | The proposed toolkit covers most common n-gram metrics and latest embedding-based metrics such as BERTScore. |
Vote’n’Rank: Revision of Benchmarking with Social Choice Theory (2023.eacl-main)
Copied to clipboard
Mark Rofin, Vladislav Mikhailov, Mikhail Florinsky, Andrey Kravchenko, Tatiana Shavrina, Elena Tutubalina, Daniel Karabekyan, Ekaterina Artemova
| Challenge: | ML benchmarks have been criticized for their construct validity, fragility of the design and task choices. |
| Approach: | They propose a framework for ranking systems in multi-task benchmarks under the principles of the social choice theory and propose 'vote'n'rank' procedures are more robust than the mean average while being able to handle missing performance scores and determine conditions under which the system becomes the winner. |
| Outcome: | The proposed framework can be utilised to draw new insights on benchmarking in several ML sub-fields and identify the best-performing systems in research and development case studies. |
SConU: Selective Conformal Uncertainty in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing frameworks fail to identify outliers that violate the exchangeability assumption, leading to unbounded miscoverage rates and unactionable prediction sets. |
| Approach: | They propose a method that implements significance tests to determine whether a given sample deviates from the uncertainty distribution of the calibration set. |
| Outcome: | The proposed approach facilitates rigorous management of miscoverage rates across single-domain and interdisciplinary contexts, and enhances the efficiency of predictions. |
SAHM: A Benchmark for Arabic Financial and Shari’ah-Compliant Reasoning (2026.acl-long)
Copied to clipboard
Rania Elbadry, Sarfraz Ahmad, Ahmed Heakl, Dani Bouch, Momina Ahsan, Muhra AlMahri, Marwa Elsaid Khalil, Yuxia Wang, Salem Lahlou, Sophia Ananiadou, Veselin Stoyanov, Jimin Huang, Xueqing Peng, Preslav Nakov, Zhuohan Xie
| Challenge: | English financial NLP has progressed rapidly through benchmarks for sentiment, document understanding, and financial question answering. |
| Approach: | They propose a document-grounded benchmark and instruction-tuning dataset for Arabic financial NLP and Shari’ah-compliant reasoning. |
| Outcome: | The proposed dataset contains 14,380 expert-verified instances spanning seven tasks . it includes financial sentiment analysis, extractive summarization, and event–cause reasoning . |